{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "initial_id",
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 1、TextSplitter的使用\n",
    "\n",
    "1、使用细节\n",
    "\n",
    "① TextSplitter作为各种具体的文档拆分器的父类\n",
    "\n",
    "② 内部定义了一些常用的属性：\n",
    "\n",
    "chunk_size: 返回块的最大尺寸，单位是字符数。默认值为4000（由长度函数测量）\n",
    "\n",
    "chunk_overlap: 相邻两个块之间的字符重叠数,避免信息在边界处被切断而丢失。默认值为200,通常会设置为chunk_size的10% - 20%。\n",
    "\n",
    "length_function: 用于测量给定块字符数的函数。默认赋值为len函数。len函数在Python中按Unicode字符计数，所以一个汉字、一个英文字母、一个符号都算一个字符。\n",
    "\n",
    "keep_separator: 是否在块中保留分隔符，默认值为False\n",
    "\n",
    "add_start_index: 如果为 `True`，则在元数据中包含块的起始索引。默认值为False\n",
    "\n",
    "strip_whitespace: 如果为 `True`，则从每个文档的开始和结束处去除空白字符。默认值为True\n",
    "\n",
    "② 内部定义的常用的方法：\n",
    "\n",
    "情况1：按照字符串进行拆分：\n",
    "\n",
    "split_text(xxx) : 传入的参数类型：字符串 ; 返回值的类型：List[str]\n",
    "\n",
    "\n",
    "create_documents(xxx) : 传入的参数类型：List[str] ; 返回值的类型：List[Document]。底层调用了split_text(xxx)\n",
    "\n",
    "情况2：按照Document对象进行拆分：\n",
    "\n",
    "split_documents(xxx) : 传入的参数类型：List[Document] ; 返回值的类型：List[Document]。底层调用了create_documents(xxx)\n",
    "\n",
    "\n",
    "\n",
    "2、Document对象 与 Str 是什么关系？\n",
    "\n",
    "文档切分器可以按照字符进行切分，也可以按照Document进行切分。其中，Str 可以理解为是Document对象的page_content属性。\n",
    "\n"
   ],
   "id": "7035cb21e0e452a5"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 2、具体的拆分器1：CharacterTextSplitter：Split by character\n",
    "\n",
    "举例1：体会：chunk_size、chunk_overlap"
   ],
   "id": "1c4f196dc46cce06"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:49:53.227391Z",
     "start_time": "2025-10-01T08:49:52.296941Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "from zipp.glob import separate\n",
    "\n",
    "# 2.示例文本\n",
    "text = \"\"\"\n",
    "LangChain 是一个用于开发由语言模型驱动的应用程序的框架的。它提供了一套工具和抽象，使开发者能够更容易地构建复杂的应用程序。\n",
    "\"\"\"\n",
    "\n",
    "# 3.定义字符分割器\n",
    "splitter = CharacterTextSplitter(\n",
    "    chunk_size=51, # 每块大小\n",
    "    chunk_overlap=7, # 块与块之间的重复字符数\n",
    "    #length_function=len,\n",
    "    separator=\"\"   # 设置为空字符串时，表示禁用分隔符\n",
    "\n",
    ")\n",
    "\n",
    "# 4.分割文本\n",
    "texts = splitter.split_text(text)\n",
    "\n",
    "# 5.打印结果\n",
    "for i,chunk in enumerate(texts):\n",
    "    print(f\"块 {i+1}:长度：{len(chunk)}\")\n",
    "    print(chunk)\n",
    "    print(\"-\" * 50)"
   ],
   "id": "1fefa909a7faaec0",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "块 1:长度：50\n",
      "LangChain 是一个用于开发由语言模型驱动的应用程序的框架的。它提供了一套工具和抽象，使开发者\n",
      "--------------------------------------------------\n",
      "块 2:长度：23\n",
      "抽象，使开发者能够更容易地构建复杂的应用程序。\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "execution_count": 1
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例2：体会separator",
   "id": "cc9787ff4be69469"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:50:35.103797Z",
     "start_time": "2025-10-01T08:50:35.096822Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "# 2.定义要分割的文本\n",
    "text = \"这是一个示例文本啊。我们将使用CharacterTextSplitter将其分割成小块。分割基于字符数。\"\n",
    "\n",
    "# text = \"\"\"\n",
    "# LangChain 是一个用于开发由语言模型。驱动的应用程序的框架的。它提供了一套工具和抽象。使开发者能够更容易地构建复杂的应用程序。\n",
    "# \"\"\"\n",
    "\n",
    "# 3.定义分割器实例\n",
    "text_splitter = CharacterTextSplitter(\n",
    "    chunk_size=30,   # 每个块的最大字符数\n",
    "    # chunk_size=43,   # 每个块的最大字符数\n",
    "    chunk_overlap=0, # 块之间的重叠字符数\n",
    "    separator=\"。\",  # 按句号分割 （分隔符优先）\n",
    ")\n",
    "\n",
    "# 4.开始分割\n",
    "chunks = text_splitter.split_text(text)\n",
    "\n",
    "# 5.打印效果\n",
    "for  i,chunk in enumerate(chunks):\n",
    "    print(f\"块 {i + 1}:长度：{len(chunk)}\")\n",
    "    print(chunk)\n",
    "    print(\"-\"*50)\n"
   ],
   "id": "6c4c869392a02587",
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Created a chunk of size 33, which is longer than the specified 30\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "块 1:长度：9\n",
      "这是一个示例文本啊\n",
      "--------------------------------------------------\n",
      "块 2:长度：33\n",
      "我们将使用CharacterTextSplitter将其分割成小块\n",
      "--------------------------------------------------\n",
      "块 3:长度：7\n",
      "分割基于字符数\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "execution_count": 2
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例3：熟悉 keep_separator、separator 以及 chunk_overlap何时生效",
   "id": "b4cf762098d1ad05"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:51:20.170702Z",
     "start_time": "2025-10-01T08:51:20.162740Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain.text_splitter import CharacterTextSplitter\n",
    "\n",
    "# 2.定义要分割的文本\n",
    "text = \"这是第一段文本。这是第二段内容。最后一段结束。\"\n",
    "\n",
    "# 3.定义字符分割器\n",
    "text_splitter = CharacterTextSplitter(\n",
    "    separator=\"。\",\n",
    "    chunk_size=20,\n",
    "    chunk_overlap=8,\n",
    "    keep_separator=True #chunk中是否保留切割符\n",
    ")\n",
    "\n",
    "# 4.分割文本\n",
    "chunks = text_splitter.split_text(text)\n",
    "\n",
    "# 5.打印结果\n",
    "for  i,chunk in enumerate(chunks):\n",
    "    print(f\"块 {i + 1}:长度：{len(chunk)}\")\n",
    "    print(chunk)\n",
    "    print(\"-\"*50)"
   ],
   "id": "2c0b3bcd72b5d06e",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "块 1:长度：15\n",
      "这是第一段文本。这是第二段内容\n",
      "--------------------------------------------------\n",
      "块 2:长度：16\n",
      "。这是第二段内容。最后一段结束。\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "execution_count": 3
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 3、具体的拆分器2：RecursiveCharacterTextSplitter：最常用\n",
    "\n",
    "举例1：使用split_text()方法演示"
   ],
   "id": "b611ca8c8e7dee6f"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:52:57.633703Z",
     "start_time": "2025-10-01T08:52:57.627405Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 2.定义RecursiveCharacterTextSplitter分割器对象\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=10,\n",
    "    chunk_overlap=0,\n",
    "    #add_start_index=True,\n",
    ")\n",
    "\n",
    "# 3.定义拆分的内容\n",
    "text=\"LangChain框架特性\\n\\n多模型集成(GPT/Claude)\\n记忆管理功能\\n链式调用设计。文档分析场景示例：需要处理PDF/Word等格式。\"\n",
    "\n",
    "# 4.拆分器分割\n",
    "paragraphs = text_splitter.split_text(text)\n",
    "\n",
    "for para in paragraphs:\n",
    "    print(para)\n",
    "    print('-------')"
   ],
   "id": "f7c1b74114c77f2f",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LangChain框\n",
      "-------\n",
      "架特性\n",
      "-------\n",
      "多模型集成(GPT\n",
      "-------\n",
      "/Claude)\n",
      "-------\n",
      "记忆管理功能\n",
      "-------\n",
      "链式调用设计。文档\n",
      "-------\n",
      "分析场景示例：需要处\n",
      "-------\n",
      "理PDF/Word等\n",
      "-------\n",
      "格式。\n",
      "-------\n"
     ]
    }
   ],
   "execution_count": 4
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例2：使用create_documents()方法演示，传入字符串列表，返回Document对象列表",
   "id": "c8881f02f3d944f3"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:53:50.174609Z",
     "start_time": "2025-10-01T08:53:50.166564Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 2.定义RecursiveCharacterTextSplitter分割器对象\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=10,\n",
    "    chunk_overlap=0,\n",
    "    add_start_index=True,\n",
    ")\n",
    "\n",
    "# 3.定义拆分的内容\n",
    "text_list = [\"LangChain框架特性\\n\\n多模型集成(GPT/Claude)\\n记忆管理功能\\n链式调用设计。文档分析场景示例：需要处理PDF/Word等格式。\"]\n",
    "\n",
    "# 4.拆分器分割\n",
    "paragraphs = text_splitter.create_documents(text_list)\n",
    "\n",
    "for para in paragraphs:\n",
    "    print(para)\n",
    "    print('-------')"
   ],
   "id": "122a9658d92acf86",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_content='LangChain框' metadata={'start_index': 0}\n",
      "-------\n",
      "page_content='架特性' metadata={'start_index': 10}\n",
      "-------\n",
      "page_content='多模型集成(GPT' metadata={'start_index': 15}\n",
      "-------\n",
      "page_content='/Claude)' metadata={'start_index': 24}\n",
      "-------\n",
      "page_content='记忆管理功能' metadata={'start_index': 33}\n",
      "-------\n",
      "page_content='链式调用设计。文档' metadata={'start_index': 40}\n",
      "-------\n",
      "page_content='分析场景示例：需要处' metadata={'start_index': 49}\n",
      "-------\n",
      "page_content='理PDF/Word等' metadata={'start_index': 59}\n",
      "-------\n",
      "page_content='格式。' metadata={'start_index': 69}\n",
      "-------\n"
     ]
    }
   ],
   "execution_count": 5
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例3：使用create_documents()方法演示，将本地文件内容加载成字符串，进行拆分",
   "id": "fb58312e39c3227d"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:56:15.230857Z",
     "start_time": "2025-10-01T08:56:15.222238Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 2.打开.txt文件\n",
    "with open(\"asset/load/08-ai.txt\", encoding=\"utf-8\") as f:\n",
    "    state_of_the_union = f.read()  #返回的是字符串\n",
    "\n",
    "print(type(state_of_the_union))  # <class 'str'>\n",
    "\n",
    "# 3.定义RecursiveCharacterTextSplitter（递归字符分割器）\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=100,\n",
    "    chunk_overlap=20,\n",
    "    #chunk_overlap=0,\n",
    "    length_function=len\n",
    ")\n",
    "\n",
    "# 4.分割文本\n",
    "texts = text_splitter.create_documents([state_of_the_union])\n",
    "print(type(texts))\n",
    "# 5.打印分割文本\n",
    "for document in texts:\n",
    "    print(f\"{type(document)}🔥{document.page_content}\")\n"
   ],
   "id": "e82e462578be0e3c",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'str'>\n",
      "<class 'list'>\n",
      "<class 'langchain_core.documents.base.Document'>🔥人工智能（AI）是什么？\n",
      "<class 'langchain_core.documents.base.Document'>🔥人工智能（Artificial\n",
      "<class 'langchain_core.documents.base.Document'>🔥Intelligence，简称AI）是指由计算机系统模拟人类智能的技术，使其能够执行通常需要人类认知能力的任务，如学习、推理、决策和语言理解。AI的核心目标是让机器具备感知环境、处理信息并自主行动的\n",
      "<class 'langchain_core.documents.base.Document'>🔥让机器具备感知环境、处理信息并自主行动的能力。\n",
      "<class 'langchain_core.documents.base.Document'>🔥1. AI的技术基础\n",
      "AI依赖多种关键技术：\n",
      "\n",
      "机器学习（ML）：通过算法让计算机从数据中学习规律，无需显式编程。例如，推荐系统通过用户历史行为预测偏好。\n",
      "<class 'langchain_core.documents.base.Document'>🔥深度学习：基于神经网络的机器学习分支，擅长处理图像、语音等复杂数据。AlphaGo击败围棋冠军便是典型案例。\n",
      "\n",
      "自然语言处理（NLP）：使计算机理解、生成人类语言，如ChatGPT的对话能力。\n",
      "<class 'langchain_core.documents.base.Document'>🔥2. AI的应用场景\n",
      "AI已渗透到日常生活和各行各业：\n",
      "\n",
      "医疗：辅助诊断（如AI分析医学影像）、药物研发加速。\n",
      "\n",
      "交通：自动驾驶汽车通过传感器和AI算法实现安全导航。\n",
      "<class 'langchain_core.documents.base.Document'>🔥金融：欺诈检测、智能投顾（如风险评估模型）。\n",
      "\n",
      "教育：个性化学习平台根据学生表现调整教学内容。\n",
      "\n",
      "3. AI的挑战与未来\n",
      "尽管前景广阔，AI仍面临问题：\n",
      "<class 'langchain_core.documents.base.Document'>🔥伦理争议：数据隐私、算法偏见（如招聘AI歧视特定群体）。\n",
      "\n",
      "就业影响：自动化可能取代部分人工岗位，但也会创造新职业。\n",
      "\n",
      "技术瓶颈：通用人工智能（AGI）尚未实现，当前AI仅擅长特定任务。\n",
      "<class 'langchain_core.documents.base.Document'>🔥未来，AI将与人类协作而非替代：医生借助AI提高诊断效率，教师利用AI定制课程。其发展需平衡技术创新与社会责任，确保技术造福全人类。\n"
     ]
    }
   ],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例4：使用split_documents()方法演示，利用PDFLoader加载文档，对文档的内容用递归切割器切割",
   "id": "b71d95dc9a98728c"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:58:11.037533Z",
     "start_time": "2025-10-01T08:58:11.017293Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain_community.document_loaders import PyPDFLoader\n",
    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
    "\n",
    "# 2.定义PyPDFLoader加载器\n",
    "loader = PyPDFLoader(\"./asset/load/02-load.pdf\")\n",
    "\n",
    "# 3.加载和切割文档对象\n",
    "docs = loader.load()   # 返回Document对象构成的list\n",
    "# print(f\"第0页：\\n{docs[0]}\")\n",
    "\n",
    "# 4.定义切割器\n",
    "text_splitter = RecursiveCharacterTextSplitter(\n",
    "    chunk_size=200,\n",
    "    #chunk_size=120,\n",
    "    chunk_overlap=0,\n",
    "    # chunk_overlap=100,\n",
    "    length_function=len,\n",
    "    add_start_index=True,\n",
    ")\n",
    "\n",
    "# 5.对pdf内容进行切割得到文档对象\n",
    "paragraphs = text_splitter.split_documents(docs)\n",
    "# paragraphs = text_splitter.create_documents([text])\n",
    "for para in paragraphs:\n",
    "    print(para.page_content)\n",
    "    print('-------')"
   ],
   "id": "ec91e04820579604",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"他的车，他的命！ 他忽然想起来，一年，二年，至少有三四年；一滴汗，两滴汗，不\n",
      "知道多少万滴汗，才挣出那辆车。从风里雨里的咬牙，从饭里茶里的自苦，才赚出那辆车。\n",
      "那辆车是他的一切挣扎与困苦的总结果与报酬，像身经百战的武士的一颗徽章。……他老想\n",
      "着远远的一辆车，可以使他自由，独立，像自己的手脚的那么一辆车。\" \n",
      " \n",
      "\"他吃，他喝，他嫖，他赌，他懒，他狡猾， 因为他没了心，他的心被人家摘了去。他\n",
      "-------\n",
      "只剩下那个高大的肉架子，等着溃烂，预备着到乱死岗子去。……体面的、要强的、好梦想\n",
      "的、利己的、个人的、健壮的、伟大的祥子，不知陪着人家送了多少回殡；不知道何时何地\n",
      "会埋起他自己来， 埋起这堕落的、 自私的、 不幸的、 社会病胎里的产儿， 个人主义的末路鬼！\n",
      "\"\n",
      "-------\n"
     ]
    }
   ],
   "execution_count": 12
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 4、具体的拆分器3：TokenTextSplitter / CharacterTextSplitter\n",
    "\n",
    "举例1：使用TokenTextSplitter"
   ],
   "id": "8248a4774d752c4b"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:58:53.846550Z",
     "start_time": "2025-10-01T08:58:53.490742Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain_text_splitters import TokenTextSplitter\n",
    "\n",
    "# 2.初始化 TokenTextSplitter\n",
    "text_splitter = TokenTextSplitter(\n",
    "    chunk_size=33,  #最大 token 数为 32\n",
    "    chunk_overlap=0, #重叠 token 数为 0\n",
    "    encoding_name=\"cl100k_base\",  # 使用 OpenAI 的编码器,将文本转换为 token 序列\n",
    "\n",
    ")\n",
    "# 3.定义文本\n",
    "text = \"人工智能是一个强大的开发框架。它支持多种语言模型和工具链。人工智能是指通过计算机程序模拟人类智能的一门科学。自20世纪50年代诞生以来，人工智能经历了多次起伏。\"\n",
    "\n",
    "# 4.开始切割\n",
    "texts = text_splitter.split_text(text)\n",
    "\n",
    "# 打印分割结果\n",
    "print(f\"原始文本被分割成了 {len(texts)} 个块:\")\n",
    "for i, chunk in enumerate(texts):\n",
    "    print(f\"块 {i+1}: 长度：{len(chunk)} 内容：{chunk}\")\n",
    "    print(\"-\" * 50)"
   ],
   "id": "da6513c5f41ee631",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原始文本被分割成了 3 个块:\n",
      "块 1: 长度：29 内容：人工智能是一个强大的开发框架。它支持多种语言模型和工具链。\n",
      "--------------------------------------------------\n",
      "块 2: 长度：32 内容：人工智能是指通过计算机程序模拟人类智能的一门科学。自20世纪50\n",
      "--------------------------------------------------\n",
      "块 3: 长度：19 内容：年代诞生以来，人工智能经历了多次起伏。\n",
      "--------------------------------------------------\n"
     ]
    }
   ],
   "execution_count": 13
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "举例2：使用CharacterTextSplitter",
   "id": "d978110ad8b94517"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T08:59:54.633691Z",
     "start_time": "2025-10-01T08:59:54.625743Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1.导入相关依赖\n",
    "from langchain_text_splitters import CharacterTextSplitter\n",
    "import tiktoken  # 用于计算Token数量\n",
    "\n",
    "\n",
    "# 2.定义通过Token切割器\n",
    "text_splitter = CharacterTextSplitter.from_tiktoken_encoder(\n",
    "    encoding_name=\"cl100k_base\", # 使用 OpenAI 的编码器\n",
    "    chunk_size=18, #设置最大的token数\n",
    "    chunk_overlap=0,\n",
    "    separator=\"。\",  # 指定中文句号为分隔符\n",
    "    keep_separator=False,  # chunk中是否保留分隔符\n",
    ")\n",
    "# 3.定义文本\n",
    "text = \"人工智能是一个强大的开发框架。它支持多种语言模型和工具链。今天天气很好，想出去踏青。但是又比较懒不想出去，怎么办\"\n",
    "\n",
    "# 4.开始切割\n",
    "texts = text_splitter.split_text(text)\n",
    "print(f\"分割后的块数: {len(texts)}\")\n",
    "\n",
    "# 5.初始化tiktoken编码器（用于Token计数）\n",
    "encoder = tiktoken.get_encoding(\"cl100k_base\")  # 确保与CharacterTextSplitter的encoding_name一致\n",
    "\n",
    "# 6.打印每个块的Token数和内容\n",
    "for i, chunk in enumerate(texts):\n",
    "    tokens = encoder.encode(chunk)  # 现在encoder已定义\n",
    "    print(f\"块 {i + 1}: {len(tokens)} Token\\n内容: {chunk}\\n\")\n"
   ],
   "id": "d6705e527e281e67",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "分割后的块数: 4\n",
      "块 1: 17 Token\n",
      "内容: 人工智能是一个强大的开发框架\n",
      "\n",
      "块 2: 14 Token\n",
      "内容: 它支持多种语言模型和工具链\n",
      "\n",
      "块 3: 18 Token\n",
      "内容: 今天天气很好，想出去踏青\n",
      "\n",
      "块 4: 21 Token\n",
      "内容: 但是又比较懒不想出去，怎么办\n",
      "\n"
     ]
    }
   ],
   "execution_count": 14
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 5、具体的拆分器4：SemanticChunker：语义分块\n",
    "\n",
    "举例："
   ],
   "id": "3cecc77a7a6ac4c1"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-10-01T09:01:30.486016Z",
     "start_time": "2025-10-01T09:01:26.389076Z"
    }
   },
   "cell_type": "code",
   "source": [
    "from langchain_experimental.text_splitter import SemanticChunker\n",
    "from langchain_openai.embeddings import OpenAIEmbeddings\n",
    "import os\n",
    "import dotenv\n",
    "\n",
    "dotenv.load_dotenv()\n",
    "\n",
    "# 加载文本\n",
    "with open(\"asset/load/09-ai1.txt\", encoding=\"utf-8\") as f:\n",
    "    state_of_the_union = f.read()  #返回字符串\n",
    "\n",
    "# 获取嵌入模型\n",
    "os.environ['OPENAI_API_KEY'] = os.getenv(\"OPENAI_API_KEY1\")\n",
    "os.environ['OPENAI_BASE_URL'] = os.getenv(\"OPENAI_BASE_URL\")\n",
    "embed_model = OpenAIEmbeddings(\n",
    "    model=\"text-embedding-3-large\"\n",
    ")\n",
    "\n",
    "# 获取切割器\n",
    "text_splitter = SemanticChunker(\n",
    "    embeddings=embed_model,\n",
    "    breakpoint_threshold_type=\"percentile\",#断点阈值类型：字面值[\"百分位数\", \"标准差\", \"四分位距\", \"梯度\"] 选其一\n",
    "    breakpoint_threshold_amount=65.0 #断点阈值数量 (极低阈值 → 高分割敏感度)\n",
    ")\n",
    "\n",
    "# 切分文档\n",
    "docs = text_splitter.create_documents(texts = [state_of_the_union])\n",
    "print(len(docs))\n",
    "for doc in docs:\n",
    "    print(f\"🔍 文档 {doc}:\")"
   ],
   "id": "9bc10f6fe0a84b72",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n",
      "🔍 文档 page_content='人工智能综述：发展、应用与未来展望\n",
      "\n",
      "摘要\n",
      "人工智能（Artificial Intelligence，AI）作为计算机科学的一个重要分支，近年来取得了突飞猛进的发展。本文综述了人工智能的发展历程、核心技术、应用领域以及未来发展趋势。通过对人工智能的定义、历史背景、主要技术（如机器学习、深度学习、自然语言处理等）的详细介绍，探讨了人工智能在医疗、金融、教育、交通等领域的应用，并分析了人工智能发展过程中面临的挑战与机遇。最后，本文对人工智能的未来发展进行了展望，提出了可能的突破方向。\n",
      "\n",
      "1. 引言\n",
      "人工智能是指通过计算机程序模拟人类智能的一门科学。自20世纪50年代诞生以来，人工智能经历了多次起伏，近年来随着计算能力的提升和大数据的普及，人工智能技术取得了显著的进展。人工智能的应用已经渗透到日常生活的方方面面，从智能手机的语音助手到自动驾驶汽车，从医疗诊断到金融分析，人工智能正在改变着人类社会的运行方式。\n",
      "\n",
      "2.':\n",
      "🔍 文档 page_content='人工智能的发展历程\n",
      "2.1 早期发展\n",
      "人工智能的概念最早可以追溯到20世纪50年代。1956年，达特茅斯会议（Dartmouth Conference）被认为是人工智能研究的正式开端。在随后的几十年里，人工智能研究经历了多次高潮与低谷。早期的研究主要集中在符号逻辑和专家系统上，但由于计算能力的限制和算法的不足，进展缓慢。\n",
      "2.2 机器学习的兴起\n",
      "20世纪90年代，随着统计学习方法的引入，机器学习逐渐成为人工智能研究的主流。支持向量机（SVM）、决策树、随机森林等算法在分类和回归任务中取得了良好的效果。这一时期，机器学习开始应用于数据挖掘、模式识别等领域。\n",
      "2.3 深度学习的突破\n",
      "2012年，深度学习在图像识别领域取得了突破性进展，标志着人工智能进入了一个新的阶段。深度学习通过多层神经网络模拟人脑的工作方式，能够自动提取特征并进行复杂的模式识别。卷积神经网络（CNN）、循环神经网络（RNN）和长短期记忆网络（LSTM）等深度学习模型在图像处理、自然语言处理、语音识别等领域取得了显著成果。\n",
      "\n",
      "3. 人工智能的核心技术\n",
      "3.1 机器学习\n",
      "机器学习是人工智能的核心技术之一，通过算法使计算机从数据中学习并做出决策。常见的机器学习算法包括监督学习、无监督学习和强化学习。监督学习通过标记数据进行训练，无监督学习则从未标记数据中寻找模式，强化学习则通过与环境交互来优化决策。\n",
      "3.2 深度学习\n",
      "深度学习是机器学习的一个子领域，通过多层神经网络进行特征提取和模式识别。深度学习在图像识别、自然语言处理、语音识别等领域取得了显著成果。常见的深度学习模型包括卷积神经网络（CNN）、循环神经网络（RNN）和长短期记忆网络（LSTM）。\n",
      "3.3 自然语言处理\n",
      "自然语言处理（NLP）是人工智能的一个重要分支，致力于使计算机能够理解和生成人类语言。NLP技术广泛应用于机器翻译、情感分析、文本分类等领域。近年来，基于深度学习的NLP模型（如BERT、GPT）在语言理解任务中取得了突破性进展。\n",
      "3.4 计算机视觉\n",
      "计算机视觉是人工智能的另一个重要分支，致力于使计算机能够理解和处理图像和视频。计算机视觉技术广泛应用于图像识别、目标检测、人脸识别等领域。深度学习模型（如CNN）在计算机视觉任务中取得了显著成果。\n",
      "\n",
      "4. 人工智能的应用领域\n",
      "4.1 医疗健康\n",
      "人工智能在医疗健康领域的应用包括疾病诊断、药物研发、个性化医疗等。通过分析医学影像和患者数据，人工智能可以帮助医生更准确地诊断疾病，提高治疗效果。\n",
      "4.2 金融\n",
      "人工智能在金融领域的应用包括风险评估、欺诈检测、算法交易等。通过分析市场数据和交易记录，人工智能可以帮助金融机构做出更明智的决策，提高运营效率。\n",
      "4.3 教育\n",
      "人工智能在教育领域的应用包括个性化学习、智能辅导、自动评分等。通过分析学生的学习数据，人工智能可以为学生提供个性化的学习建议，提高学习效果。\n",
      "4.4 交通\n",
      "人工智能在交通领域的应用包括自动驾驶、交通管理、智能导航等。通过分析交通数据和路况信息，人工智能可以帮助优化交通流量，提高交通安全。\n",
      "\n",
      "5. 人工智能的挑战与机遇\n",
      "5.1 挑战\n",
      "人工智能发展过程中面临的主要挑战包括数据隐私、算法偏见、安全性问题等。数据隐私问题涉及到个人数据的收集和使用，算法偏见问题则涉及到算法的公平性和透明度，安全性问题则涉及到人工智能系统的可靠性和稳定性。\n",
      "5.2 机遇\n",
      "尽管面临挑战，人工智能的发展也带来了巨大的机遇。人工智能技术的进步将推动各行各业的创新，提高生产效率，改善生活质量。未来，人工智能有望在更多领域取得突破，为人类社会带来更多的便利和福祉。\n",
      "\n",
      "6.':\n",
      "🔍 文档 page_content='未来展望\n",
      "6.1 技术突破\n",
      "未来，人工智能技术有望在以下几个方面取得突破：一是算法的优化和创新，提高模型的效率和准确性；二是计算能力的提升，支持更复杂的模型和更大规模的数据处理；三是跨学科研究的深入，推动人工智能与其他领域的融合。\n",
      "6.2 应用拓展\n",
      "随着技术的进步，人工智能的应用领域将进一步拓展。未来，人工智能有望在更多领域发挥重要作用，如环境保护、能源管理、智能制造等。人工智能将成为推动社会进步的重要力量。\n",
      "\n",
      "7.':\n",
      "🔍 文档 page_content='结论\n",
      "人工智能作为一门快速发展的科学，正在改变着人类社会的运行方式。通过不断的技术创新和应用拓展，人工智能将为人类社会带来更多的便利和福祉。然而，人工智能的发展也面临着诸多挑战，需要社会各界共同努力，推动人工智能的健康发展。':\n"
     ]
    }
   ],
   "execution_count": 15
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
